ci: fail every lane that selects tests and executes none - #692

What this is

A pre-release audit finding on the CI lanes, and the fix for the whole class rather than the two instances that were reported.

cargo test with a filter that matches nothing prints running 0 tests, then test result: ok, and exits 0. A lane in that state looks exactly like a lane that proved something. The repo already knew this: #659 added a count to the aerorsync live lane, and ftp-mlsd.yml carries the same check on its listing-timeout step. Two instances had the check. Eleven siblings did not.

One of those lanes was already dead, and this is the evidence

delta-sync-integration.yml selected product_path_falls_through_silently_when_session_not_eligible. That test was renamed on 2026-05-16 in bb20f6a6c, when password-backed SFTP became eligible for the native leg with the host key pinned and the assertion inverted. The workflow line was never updated.

So since that day the job named "SFTP password-only fixture + fallback contract" has built a Docker fixture, waited for SSH, executed zero tests, and reported success. From run 33434171154 on main, tonight:

running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 17 filtered out; finished in 0.00s

The step now names product_path_uses_native_delta_for_password_sftp_with_pinned_host_key, which is where that contract lives today, and asserts that it ran.

The fix

One script, scripts/assert-tests-ran.sh, called from fourteen places, instead of fourteen copies of a shell snippet whose subtleties decay independently. The subtleties are real and are now written down once:

• set -o pipefail at the call site, or the step's status is tee's and not cargo's, which is the same blindness this exists to remove nested inside its own cure.
• no bare assignment from a pipeline under bash -e, which is what a step without an explicit shell: runs, and which shell: bash does not fix, since GitHub expands that to bash -eo pipefail.
• an empty or missing log is reported as its own failure rather than counted as zero: a run that died before libtest printed anything is not a run of zero tests.

=N means exactly N and is used where a step names one test, because a second test arriving there changes what the step does and should be said out loud. A plain N means at least N and is used where a step collects a module or a binary, because such a suite is meant to grow and an exact match would go red the day someone adds a legitimate test, which usually gets answered by deleting the check.

The numbers are measurements, not guesses

Each floor is what the lane actually executed on a real green run, named in the comment beside it, and each was also reproduced locally:

Shown red before being trusted

A count that has never been seen to fail is the thing being fixed here, so neither check was committed until it failed on purpose.

MLSD. With the two #[ignore] attributes removed from integration_ftp_mlsd.rs, which is the realistic way this breaks, since removing them looks like an improvement:

• today's line: running 0 tests, test result: ok, step exit 0, green.
• the same run with the assertion: step exit 1, ::error::MLSD listing tests: executed 0 test(s), expected at least 2.

delta-sync fallback. Running the line as it stands on main, against the real test binary: exit 0 and zero tests; with the assertion, exit 1.

The script itself, under bash -e, which is the shell a GitHub step gets, against twelve inputs: at the floor, above it, below it, zero, exactly one, more than one where one was promised, two binaries summed, a FAILED result line, an empty log, a missing log, and two malformed argument sets (which exit 2, to keep "you called me wrong" apart from "the gate failed").

What was NOT verified

• No lane was run end to end here. These jobs need Docker fixtures, a real rsync server and a runner. What was verified locally is the selection and the count for every step, plus the assertion's behaviour on real cargo output.
• The 677 floor was reconciled between CI and local: 692 tests match the aerorsync substring, of which 15 are ignored. Two of the 692 are outside the aerorsync:: module and match by substring, which is why an anchored count gives 690 and the lane reports 692.
• The nightly numbers come from run 33377726369, the most recent successful nightly, not from tonight's tree. If a test landed in transfer_dag:: since, the floor is conservative, which is the safe direction.

The instance deliberately left

tests/gtc/parity_harness.sh runs a per-cell test name and has the same shape. It is a developer harness rather than a gate, it is not invoked by any workflow, it has its own exit-code handling, and it already parses the log downstream in a way that notices an empty run. It is named here so the class is closed on paper as well as in the tree.

build.yml runs cargo test with no filter and cannot select nothing, so it needs no assertion.

Risk

No Rust changed, so the fmt, clippy and test gate is the one main already carries.

The floors can only go red by executing FEWER tests than the lane executes today. Adding tests keeps every lane green. If tests are removed deliberately, the lane goes red with a message that says to lower the number in that same commit.

.gitignore carries /scripts/* with per-file exceptions, so the new script needs its own exception line or it would never have been committed and every guarded lane would fail on a missing file. That line is in this PR.

Summary by CodeRabbit

• Bug Fixes
  • CI checks now detect when filtered test commands run fewer tests than expected—or none at all—preventing misleading successful builds.
  • Test validation now covers Aerorsync, delta-sync, FTP, telemetry, and capture workflows.
  • Test output is retained in logs to improve failure diagnosis.
• Chores
  • Added shared validation for minimum and exact test counts across automated checks.

Merge record

Merged at zero incomplete check runs, read from the check-runs API on the head commit rather than from this page, where a superseded cancelled run reads as red.

CodeRabbit reviewed earlier commits on this branch and raised one Major finding, on a tee to a fixed name under /tmp in the two capture harnesses, which is fixed in 226981fa after being REPRODUCED in a test TMPDIR rather than deduced from the rule. The incremental pass over 7204d9dc..226981fa was then rate limited, so the head as merged carries a human reading only.

The change this pull request makes is to the instruments themselves, so it deserves the plainest possible statement of what was and was not established. Established: each of the fourteen call sites was exercised against real cargo output, the thresholds are what a named green run actually executed rather than estimates, and both gates were watched going RED with the condition made true. Not established: no lane was run end to end here, so what is proven is the selection, the counting and the behaviour of the assertion, not that every lane still passes in CI. That is what the CI run on this pull request is for, and it is green.